1. Field of the Invention
The present invention relates to a picture processing apparatus aiming at processing a plurality of pictures. In particular, the invention relates to a picture processing apparatus for detecting a mismatch of images within two pictures as well as to a picture processing apparatus for obtaining a composite picture high resolution or a picture of wide angle/wide range having a greater number of pixels than a picture obtained by an image pickup device by one time of image pickup operation. The invention also relates to a picture processing apparatus for compensating a density value of pixels of each picture so that the overall lightness of two or more pictures is equalized, and for correctly compensating color tone of the composite picture.
2. Description of Related Art
A picture processing apparatus used as a so-called scanner is provided with a one-dimensional image pickup device like a CCD line sensor in which a plurality of photo-receiving areas are arrayed in a row. This picture processing apparatus produces an output picture, by picking up images of a subject at a plurality of positions while moving the image pickup device with respect to the subject in a direction orthogonal to a direction in which the photo-receiving areas are arrayed on a two-dimensional plane. The apparatus then composites the pictures obtained by the respective image pickup operations such that they are arrayed on the two-dimensional plane in a relationship of a position identical to a position of the image pickup device in picking up each image, to produce a single output picture. As compared to a picture obtained directly by the image pickup device, though the number of pixels arrayed in the same direction as a moving direction of the image pickup device increases in the picture obtained by this apparatus, the number of pixels in a direction orthogonal to the moving direction is no different. As a method for increasing the number of pixels of the picture in the above-mentioned orthogonal direction, there has been a method of compositing the plurality of above-mentioned output pictures by arraying along the orthogonal direction.
As a first prior art technology using the above-mentioned method, there may be cited a method disclosed in Japanese Unexamined Patent Publication JP-A 63-64180 (1988). According to an apparatus using the picture image compositing method disclosed in this publication, a one-dimensional image pickup device is provided within a manually movable hand scanner. According to this method, at first, an original picture to be input is divided into areas of a size readable by one time of scan by using the hand scanner, and a picture of each area is obtained by picking up its image by moving the hand scanner along a center axial line of the areas with the scanner contacted on the surface of the original picture. The picture of each area is equivalent to the output picture of the above-mentioned apparatus using the one-dimensional image pickup device and each image has an overlapping area where the same part of the original picture has been read. Next, the overlapping areas of the respective areas are matched by way of the template matching method to find a positional mismatch of the images having the same shape and the same distribution of brightness between the respective images. Finally, based on the positional mismatch of the images, each picture is moved in parallel and superimposed so that those images overlap to obtain a composite picture image.
In this picture image compositing method, since an operator of the apparatus manually moves the hand scanner containing the image pickup device, an unintentional movement of the hands of the operator may be applied to the hand scanner when the scanner is moved. A relative position between the image pickup device within the hand scanner and the center axial line of the above-mentioned areas of the original picture to be input may also incline for the same reason. Further, a pair of rollers are provided within a part of the hand scanner where it contacts with the original picture so that the hand scanner moves smoothly on the surface of the original picture, and the moving speed of the hand scanner may differ between parts thereof near one roller and near the other roller, when the smoothness of the pair of rollers is different from each other. When these things happen, a distortion of an image including a mismatch of the image in a direction different from a direction of the parallel movement and a partial compression of the image occurs within the pictures. Because the images of the subject in the plurality of pictures having such a distortion are not smoothly joined just by horizontally moving the pictures, the image distorts in the composited picture.
In order to compensate this mismatch, the applicant of the present invention has proposed, in Japanese Unexamined Patent Publication JP-A 5-260264 (1993), a technology for obtaining an output picture composed of a more number of pixels than a number of photo-receiving areas of an image pickup device by using a two-dimensional image pickup device in which photo-receiving areas are arrayed in a matrix. This technology will be explained below as a second prior art technology. According to a picture processing apparatus disclosed in this publication, an operator holds means for inputting an image, i.e. a so-called CCD imaging apparatus, and shoots a subject while horizontally moving the imaging apparatus at a position distant from the subject by a predetermined distance to obtain a plurality of pictures. A processing circuit within the above-mentioned picture processing apparatus matches a new picture with a composite picture by way of, e.g., the block matching method every time when the new picture is obtained to find a positional mismatch of the same image within both the pictures. Next, based on this mismatch, the new picture is transformed and the transformed image is finally composited with the composite picture to obtain a new composite picture.
Because the image inputting means of this picture processing apparatus is supported only by the hands of the operator of the apparatus and is held in air, a relative position between the subject and the image pickup device is liable to shift. Due to that, a distortion of the image including a mismatch of the image in a direction other than a direction in which the image inputting means is moved horizontally, an inclination of the image and a different magnification factor, occurs between the images in the two pictures to be composited. Although the picture processing apparatus composites the pictures while compensating this distortion of the image when compositing, all distortions may not be compensated and some distortion remain due to, e.g., a matching error and a compensation error which will be described later. Further, arithmetic processing for this compensation is complicated and a processing amount of the compensation increases.
As a third prior art technology, there may be cited a technology disclosed in Japanese Unexamined Patent Publication JP-A 4-314263 (1992). A picture processing apparatus disclosed in this publication first extracts features such as an edge of an image from data sets of a plurality of pictures to be composited which have been input from a scanner. Then, it obtains a composite picture by generating address information for joining the picture data sets based on the features and by joining the plurality of data sets at positions specified by the address information.
The applicant of the present invention has also proposed a technology for automatically compositing and copying images such as maps printed across a plurality of pages in Japanese Unexamined Patent Publication JP-A 6-181518 (1994). This technology will be explained below as a fourth prior art technology. According to a picture processing apparatus of the disclosure, a predefined symbol is previously marked at a part of a plurality of originals to be read containing an overlapping image. Then, a plurality of original data sets are created at first by reading those originals individually by a CCD sensor constituting a reading section of a copying machine. Next, a combination for joining the respective original data sets and joints within the original data sets are recognized by extracting the above-mentioned symbol by retrieving an outer peripheral part of each original data set. Finally, they are composited by rearranging the original data sets of the recognized combination so that the joints face to each other and by aligning them so that the images at the joints coincide.
In addition to the picture processing apparatus described in JP-A 4-314263, U.S. Pat. No. 5,465,163, i.e. a fifth prior art technology, has disclosed a picture processing apparatus which performs the following process. When a size of an original is larger than a size readable at one time by the picture processing apparatus, a plurality of picture areas which overlap each other are set on the original and a scanner of the picture processing apparatus reads the respective picture areas. Patterns of a plurality of picture data sets to be composited obtained as a result are then matched and based on address information obtained as a result, the plurality of picture data sets are joined.
U.S. Pat. No. 5,465,163 has also disclosed a picture processing apparatus which performs the following process. When the size of the original is larger than the readable size, an operator puts marks at the original with a color marker in advance. Then, a plurality of picture areas each containing the above-mentioned mark are created on the original and the scanner reads the respective picture areas. Further, when a plurality of originals which overlap partially from each other are to be read, the operator puts marks at the above-mentioned part in each original with the color marker and then the scanner reads the plurality of originals. Patterns of the marks respectively imaged on the pictures represented by the plurality of picture data sets to be composited obtained as a result are matched and based on address information obtained as a result, the plurality of picture data sets are joined.
Because a scanner and a reading section of a copying machine are used as a reading device in the third through fifth prior art picture processing apparatuses, and the subject and the reading device are fixed respectively at predetermined positions, so that the positional relationship between both the is always kept constant. Accordingly, the above-mentioned imaginal distortion which is otherwise caused by the mismatch of the positional relationship barely occurs in the picture data set and the original data set. Therefore, no means for correcting the imaginal distortion is provided in the above-mentioned picture processing apparatuses. Accordingly, the imaginal distortion may remain in the composited picture when the pictures obtained by the same method as the first and second prior art technologies are composited by the same method as the third and fourth prior art technologies.
By the way, the first and second prior art picture processing apparatuses find the imaginal mismatch by way of pattern matching. In the pattern matching method, an operation of superimposing a pair of pictures to calculate a correlation between the pair of pictures at that time is repeated by a plurality of times while the position where they are superimposed is changed and a positional mismatch of the pictures when the pictures are superimposed when the correlation is maximum is detected as an imaginal mismatch. This pattern matching method includes a typical point matching method and a block matching method.
The typical point matching method is a method that a plurality of pixels discretely arrayed and at predefined positions among pixels of one picture of a pair of pictures are set as typical points in advance, differences of brightness between each typical point and a pixel of the other picture overlapping with that typical point are cumulatively added, and its sum is obtained as a value representing the correlation. However, a difference of magnitude of the correlation cannot be obtained by the typical point matching method when the brightness of the plurality of pixels where the respective typical points are overlapped one by one does not change when the position for superimposing the pair of pictures is moved because the value representing the correlation does not change. Accordingly, when a monochromic picture like a document picture is to be processed, it becomes difficult to precisely detect an imaginal mismatch because the change in brightness is small at all of the typical points and it becomes hard to detect the position where the correlation is maximum.
The pattern matching method is a method that a reference area having a predefined size is set within one picture of a pair of pictures and a difference of brightness between each pixel within the reference area and a pixel in the other picture which overlaps with that pixel are cumulatively added when the pair of pictures are superimposed to obtain its sum as a value representing the correlation. When an image whose shape and whose distribution of brightness are equal to those of the image within the reference area is contained in the other picture, the sum turns out to be zero and the correlation is assumed to be maximum when the pair of pictures are superimposed so that the images overlap.
When the pair of pictures are pictures obtained by the methods described in the first and second prior art technologies at this time, the shape and distribution of brightness of the images representing the same subject are different in the pair of pictures due to the aforementioned imaginal distortion. Suppose that an image "A" in FIG. 43A is contained in one picture and an image "A" in FIG. 43B is contained in the other picture for example. Although both the images represent the same character "A", their shape and brightness are different. Even if the pair of pictures are superimposed so that their centers coincide and the images almost overlap, there arise parts which do not overlap each other within the pictures. Because values of brightness of overlapping pixels do not coincide in these parts, the aforementioned sum obtained at this position may be greater than a sum obtained at position other than the above-mentioned position. Accordingly, it becomes difficult to detect the imaginal mismatch by the magnitude of the sums and hence to superimpose the pair of pictures reliably in compositing them.
The block matching method is a method that a difference of brightness between all pixels in the reference area and pixels in the other picture overlapping with each pixel in the reference area is found per pixel to find its sum. Therefore, because there are a large number of pixels whose difference of brightness is to be found in one time of computation of the correlation, it requires an extremely large amount of processing for obtaining the imaginal mismatch and hence takes much time for the arithmetic processing. For instance, when the mismatch of the pair of pictures is to be detected to detect the unintentional movement of the hand by using the block matching method, it is necessary to increase a processing speed of an arithmetic processing circuit for performing the block matching to the extreme.
Further, as another matching method, there is a method that a plurality of feature points within one picture are extracted in advance, a correlation between the feature point and a point within the other picture is found per feature point, and a mismatch is detected by a difference of magnitude of the sum of the correlation. Such a matching method will be referred to as a feature point matching method hereinafter. For instance, an edge of the image shown in FIG. 43A is extracted and a plurality of feature points are extracted from among points on the edge as shown in FIG. 44A. In FIG. 44A, a thin line represents the edge of the image and black square marks represent the feature points. When a mismatch between the image of FIG. 43A and the image of FIG. 44B is to be found by the feature point matching method, the image of FIG. 44B and the feature points of FIG. 44A are overlapped to individually find the correlation between each feature point and a point overlapping with the feature point within the image of FIG. 44B. The correlation is maximized when all of the overlapping points are the points on the edge of the image of FIG. 44B.
When the pair of pictures have the imaginal distortion due to the above-mentioned reason in this case, the shape of the edge of the images within the pair of pictures are not congruent. Therefore, even when the pair of pictures are superimposed so that one feature point overlaps with a point corresponding to that feature point within the edge of the image, other feature points do not overlap with points corresponding to that feature points. The corresponding point is a point representing the same part as the part of the subject represented by the feature point in the image within the other picture among the images representing the same subject as the subject represented by the image of one picture within the other picture.
Accordingly, the correlation in this state may be smaller than the correlation in a state that the pair of pictures are superimposed as described above when the images are congruent and may not be maximized among a plurality of correlations found by changing the superimposing state one by one. Therefore, when the mismatch is found based on this correlation, an error of the mismatch may increase because the correlation in a state that the images do not overlap at all is maximized and the mismatch of the pictures at that time is detected as an imaginal mismatch. That is, the precision of the matching drops. Those are the problematic points caused by the imaginal mismatch of the plurality of pictures in compositing them.
Further, the prior art picture processing apparatus finds an overlapping area where a pair of pictures overlap each other at first for compositing both the pictures represented by picture signals. It then composites the picture signals each representing each picture so that the pair of pictures are arranged so that the overlapping areas overlap to generate a picture signal representing a composite picture.
In order to obtain the pair of pictures, one imaging apparatus comprising a CCD image sensor and an AGC (Auto Gain Control) circuit for example is used and a subject is shot twice in succession. Because the AGC circuit automatically changes a gain of the CCD image sensor corresponding to shooting conditions such as brightness around the subject at this time, overall lightness of each picture, i.e. density and tone of each picture, may differ from each other. This happens remarkably when the shooting is performed under a light source such as a fluorescent lamp whose brightness changes continuously. The whole lightness of each picture may also differ from each other due to characteristic differences of image pickup devices of respective imaging apparatuss when the pair of pictures are to be obtained by shooting the subject with the plurality of imaging apparatuss at the same time.
When such pair of pictures are composited, density of pixels in the composite picture may change sharply at the part according to the joint of both the pictures in the composite picture and thereby a boarder line of the density and tone may appear within the composite picture. Then, in order to smooth the changes of density of pixels within the composite picture, a gradation process, for example, is implemented in compositing picture signals. In deciding the density of each pixel in the composite picture by the gradation process, a ratio of each picture contributing to the decision of the density is changed stepwise or continuously corresponding to a distance between the part corresponding to the joint within the composite picture and the pixel to be processed. The gradation process can be used when a difference of the overall lightness between both the pictures is small, and the change of density of the composite picture is hardly smoothed when this difference is large even when the gradation process is used.
There are also methods of adjusting the overall lightness of a pair of pictures before compositing picture signals in order to smoothly change the density of pixels of the composite picture. As the density compensating methods for adjusting the overall lightness of the pair of pictures, there have been known methods of histogram coincidence and linear density transformation.
The histogram coincidence method is a method that a histogram of values of density of pixels in an overlapping area of each picture is respectively found, and referring to the found histogram, the density of pixels of the whole of each picture is converted so that the histograms of both the overlapping areas coincide. This histogram coincidence method is described in "Handbook on Image Analysis" (University of Tokyo Press, Editorial Supervisors: Mikio Takagi & Yohisa Simoda), pp., 463, 478 and 479.
As a prior art technology using the histogram coincidence method, there may be cited a picture processing method and a picture processing system disclosed in Japanese Unexamined Patent Publication JP-A 5-342344 (1993). According to this picture processing method, a plurality of color components of two color pictures to be pasted, i.e. a plurality of data sets representing their hue, saturation and lightness, are generated at first and then an area common to both the color pictures, i.e. an overlapping area, is designated. Next, a histogram of each color component in the common area is found, a histogram of each color component of one color picture is transformed such that the histogram of each color component of one color picture coincides with a histogram of each color component of the other color picture, and colors of pixels of the color picture are corrected according to the content of the transformation.
Such picture processing method requires to perform a subdivision of a density value, i.e. to distribute a plurality of pixels having the same density or color component each other to a plurality of other values of density or color components, so that the histograms of both the pictures coincide. This subdivision is performed by distributing the plurality of pixels of the same density by random number or by ranking the plurality of pixels of the same density according to an average density value of the plurality of pixels around those pixels to distribute according to that ranking for example. The process of the density compensating method using the histogram coincidence method is complicated because this subdivision has to be performed.
In the linear density transformation method, it is supposed that a linear transformation holds between density of a pair of corresponding pixels in an overlapping area of a pair of pictures, a density value PL each representing the density of all pixels of one picture of the pair of pictures is transformed to a density value PL* based on the following Expression (1). Coefficients a and b in the expression are decided by the method of least square based on the values of density of the above-mentioned plurality of pairs of pixels. This linear density interpolation method is described in "Handbook on Image Analysis" pp., 463 and 464. EQU PL*=aPL*+b (1)
When the pair of pictures are to be obtained by the above-mentioned imaging apparatus, an image of a subject on the picture image may distort partially due to a lens and other factors of the imaging apparatus. Further, due to an error in finding the overlapping area, the found overlapping area may deviate from an area where the pictures actually overlap. Thereby, the correspondence between the pixels in the overlapping areas may shift. Due to that, it becomes difficult to correctly estimate the coefficients in Expression (1) even when the method of least square is used.
Still more, although the density compensating methods using the histogram coincidence method and the linear density transformation method allow the overall lightness of two or three pictures to be adjusted, it is difficult to adjust the overall lightness of 10 or more pictures for example due to the following reason.
For example, when a plurality of pictures are to be obtained by shooting a subject by plural times while a shooting range of the imaging apparatus is moved, reflection of the surface of or shadow of the subject may enter within the shooting range. When the shooting range is moved so that the positional relationship among the subject, the imaging apparatus and a light source barely changes, a moving speed of the shooting range is greater than a moving speed of position of the reflection and shadow. Due to that, the reflection and shadow are present almost at the same position in each picture. Further, in general, lightness at the peripheral part of the picture may be darkened as compared to that at the center part, due to dispersion of sensitivity of a plurality of photo-receiving areas of the image pickup device and due to shading of the lens and the image pickup device in general.
A position of the overlapping area with respect to an outer frame of one picture of a pair of pictures shifts from a position of the overlapping area with respect to an outer frame of the other picture, according to the moving speed of the shooting range. Due to that, there is a case where the reflection exists in the overlapping area of one picture and the shadow exists in the overlapping area of the other picture. Further, there is a case where the overlapping area of one picture is located at the center of the picture and the overlapping area of the other picture is located at the peripheral part of the picture. When these two problems arise, the distribution of density of the pixels in both the overlapping areas differ even though the same subject is imaged in the overlapping area of both the pictures.
In compensating the overall lightness of a plurality of pictures, the picture processing apparatus selects two pictures having an overlapping area among the plurality of pictures at first and corrects the values of density of pixels of these two pictures by using the above-mentioned density compensating method. Next, it selects one picture of the two corrected pictures and another picture having an overlapping area with that one picture and corrects the lightness so that the overall lightness of the other picture is equalized with the overall lightness of the one picture by using the above-mentioned density compensating method. It further performs the latter compensating process repeatedly to the remaining pictures one by one.
When the plurality of pictures have either one of the above-mentioned two problems at this time, the picture processing apparatus determines that the distribution of values of density of the overlapping area of one picture is different from that of the other picture. It then corrects the density value of the pixels of the other picture to be lighter or darker than the density value of the pixels of one picture even when the overall lightness of the pictures are almost the same. Accordingly, when this correction is repeatedly performed, the effect of the reflection, shadow and shading becomes greater in the pictures corrected later, so that the overall lightness of the other corrected picture is darkened or lightened gradually as compared to the overall lightness of the initially selected picture. Thereby, the overall lightness of the plurality of pictures do not coincide after the correction.
Although the effect of the above-mentioned problem is small when the number of the pictures to be processed is two or three, the more the number of the pictures becomes, the greater the effect of the above-mentioned problem becomes and the mismatch of the overall lightness may be clearly observed even by human eyes when the number of pictures to be processed is around ten. Due to that, when these pictures are composited, a joint appears in the composited picture because the density and tone of the composite picture vary partially.
As one prior art technology concerning to the transformation of values of density for compositing and processing pictures, there may be cited a picture compositing apparatus disclosed in Japanese Unexamined Patent Publication JP-A 5-3538 (1993). This picture compositing apparatus forms a composite picture by optically scanning two pictures to read as a plurality of kinds of color signals and by mixing the respective color signals so that those two pictures overlap within the same area in a digital copying machine and the like. At this time, it performs a compensating process such as correction of density value of the composite picture based on the color signals of one or more color among the color signals of one picture.
Because this compensating process is performed in order not to saturate the density value of pixels of the composite picture and to clarify the picture, its purpose is different from that of the process for compensating the density value in the process for compositing pictures by joining two pictures as described above. Further, while the density value of the two pictures is either cut into a half uniformly or is changed variably in performing the compensating process, it is not described how its correction factor should be defined. Accordingly, it is difficult to adjust the overall lightness of two pictures precisely when the compensating process is performed in the picture compositing process for joining two pictures by using the above-mentioned compensating process.
Further, as another prior art technology concerning the transformation of values of density of pictures, there may be cited a picture processing apparatus disclosed in Japanese Unexamined Patent Publication JP-A 6-301775 (1994). This picture processing apparatus finds a variance of a value of pixel (density value) of each pixel of an input picture to be processed, and finds an average value and a maximum value of those variances. The pixels are binarized based on the variance when the average value is almost equal to the maximum value and the pixels of a picture obtained by removing a background from the input picture are binarized when they are not equal. This picture processing method is performed to favorably and readily binarize the input picture in which lines, shading and density are mixed and the method is different from the density compensating process for adjusting the overall lightness of two pictures. Therefore, it is difficult to adjust the overall lightness of the two pictures by using this picture processing method.
Further, the above-mentioned imaging apparatus may be provided with an AWB (Auto White Balance) circuit of a picture signal detecting system for automatically compensating color tone of the whole picture based only on picture signals. However, it is difficult to correctly correct the tone by the AWB circuit when the color of pixels composing the picture is almost one and the same. In particular, it is difficult to correctly correct the tone of pictures to be composited because only a part of a subject is often imaged on the pictures and the color of the pixels is liable to be one-sided to a specific color because a magnification factor in shooting them is often large as compared to pictures in other uses.
Still more, as a prior art technology using the white balance correction, there may be cited a imaging apparatus disclosed in Japanese Unexamined Patent Publication JP-A 6-261333 (1994). This imaging apparatus obtains a plurality of pictures to be composited by dividing a subject into a plurality of areas and by shooting the respective areas. It then integrates characteristics of a picture signal of each picture and based on an integrated value thereof, corrects the white balance and density value of the picture signal. Because the pictures represented by the respective picture signals are what the respective different area are shot, values of density and color tones of pixels on both the sides of a boarder which corresponds to the joint of the pictures within the composite picture may not coincide when the pictures corrected by the white balance correction are composited. These are the problematic points caused by the difference of the overall lightness among the plurality of pictures in composing the pictures.